Pareto models for discriminative multiclass linear dimensionality reduction

Linear Discriminant Analysis (LDA) is a popular tool for multiclass discriminative dimensionality reduction. However, LDA suffers from two major drawbacks: (i) it only optimizes the Bayes error for two-class problems where each class is a unimodal Gaussian with different mean, but both classes have equal full rank covariance matrices, and (ii) the multiclass extension does not maximize each pairwise distance between the classes, but rather maximizes the sum of all these pairwise distances. This typically results in serious overlapping in the projected space between classes that are close to each other in the input space, and degrades classification performance. To solve these two problems, this paper proposes Pareto Discriminant Analysis (PARDA). First, PARDA explicitly models each class as a multidimensional Gaussian distribution (or as a mixture of Gaussians) with a full rank covariance matrix. Second, PARDA decomposes the multiclass problem to a set of pairwise objective functions representing the pairwise distance between different classes. Unlike existing extensions of Fisher discriminant analysis (FDA) to multiclass problems that typically maximize the sum of all pairwise distances, PARDA maximizes each pairwise distance, guided by an ultimate objective to maximally separate all class means, while minimizing the class overlap (due to class Corresponding Author Email addresses: [email protected] (Karim T. Abou–Moustafa), [email protected] (Fernando De La Torre), [email protected] (Frank P. Ferrie) Preprint submitted to Pattern Recognition August 14, 2014 spread) in the lower dimensional space. Solving PARDA is a multiobjective optimization problem – simultaneously optimizing more than one, possibly conflicting, objective functions – and the resulting solution is known to be “Pareto optimal”. Experimental results on various data sets show promising results in favour of PARDA when compared with standard and modern multiclass extensions of LDA.